Vehicle and trailer maneuver assist system

ABSTRACT

A trailer assist system for a vehicle includes a plurality of sensors disposed at a vehicle, a trailer sensor disposed at a rear portion of a trailer and having a field of sensing at least rearward of the trailer, and a control having a processor operable to process data captured by the sensors. A display screen displays an area rearward of the trailer. Responsive to a user input selecting a target area present in the displayed image, the control, via processing of data captured by the sensors, is operable to determine a path of travel for the vehicle and trailer to follow to maneuver the vehicle and trailer toward a target location represented by the selected displayed target area. The control, responsive to determination of the path of travel, controls at least steering of the vehicle to maneuver the vehicle and trailer along the determined path of travel.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims the filing benefits of U.S. provisionalapplication Ser. No. 62/667,729, filed May 7, 2018, U.S. provisionalapplication Ser. No. 62/569,658, filed Oct. 9, 2017, and U.S.provisional application Ser. No. 62/549,989, filed Aug. 25, 2017, whichare hereby incorporated herein by reference in their entireties.

FIELD OF THE INVENTION

The present invention relates generally to a vehicle vision system for avehicle and, more particularly, to a vehicle vision system that utilizesone or more cameras at a vehicle.

BACKGROUND OF THE INVENTION

Use of imaging sensors in vehicle imaging systems is common and known.Examples of such known systems are described in U.S. Pat. Nos.5,949,331; 5,670,935 and/or 5,550,677, which are hereby incorporatedherein by reference in their entireties. It is also known to use camerasto assist in hitching a vehicle to a trailer and/or in determining atrailer angle of a trailer relative to a vehicle. Examples of such knownsystems are described in U.S. Pat. Nos. 9,085,261 and/or 6,690,268,which are hereby incorporated herein by reference in their entireties

SUMMARY OF THE INVENTION

The present invention provides a trailer maneuver assist system for avehicle that utilizes one or more cameras (preferably one or more CMOScameras) to capture image data representative of images exterior andrearward of the vehicle and trailer towed by the vehicle, and controlsthe vehicle to reverse the vehicle and trailer along a determined pathof travel and toward a target location. The system displays bird's-eyeview or surround view images of the vehicle and trailer and arearearward of the trailer and the user selects a target location. Thesystem determines an appropriate (obstacle free) path for the vehicleand trailer to follow to maneuver the trailer toward the selected targetlocation. The system provides control algorithms to assist maneuveringthe vehicle and trailer without the need for manual steering,acceleration or brake input by the driver.

These and other objects, advantages, purposes and features of thepresent invention will become apparent upon review of the followingspecification in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a vehicle with a trailer maneuver assist systemthat is operable to control or guide the vehicle and to reverse thevehicle and trailer toward and into a selected location in accordancewith the present invention;

FIG. 2 is a block diagram of the functions of the trailer maneuverassist system of the present invention;

FIG. 3 is a block diagram of the algorithm flow of the trailer maneuverassist system of the present invention;

FIG. 4 is a view of the scene map and top view of the vehicle andtrailer as displayed by a display screen to allow the user to select thetargeted location;

FIG. 5 is a plan view of the vehicle and trailer showing pixel motionvectors for the trailer camera and the vehicle multi-camera system;

FIG. 6 is a plan view of a vehicle with another trailer maneuver assistsystem that is operable to control or guide the vehicle and to reversethe vehicle and trailer toward and into a selected location inaccordance with the present invention;

FIG. 6A is another plan view of a vehicle with another trailer maneuverassist system that is operable to control or guide the vehicle and toreverse the vehicle and trailer toward and into a selected location inaccordance with the present invention;

FIG. 7 is a block diagram of the functions of another trailer maneuverassist system of the present invention;

FIG. 8 is a block diagram of the algorithm flow of the trailer maneuverassist system of the present invention;

FIG. 9 is an image of a navigation screen showing a determined path froma current location of the vehicle and trailer to a user-selecteddestination;

FIG. 10 is a side elevation of the vehicle and trailer showing thesystem tracking a target in accordance with the present invention;

FIG. 11 is a view of a scene map and top view of the vehicle and traileras displayed by the display screen to allow the user to select currentand new targeted locations;

FIG. 12 is a view of another scene map and top view of the vehicle andtrailer as displayed by the display screen to allow the user to selectcurrent and new targeted locations;

FIG. 13 is a view of another scene map and top view of the vehicle andtrailer as displayed by the display screen to allow the user to select alocation and orientation; and

FIG. 14 is a view of another scene map and top view of the vehicle andtrailer as displayed by the display screen to allow the user to select anew location and orientation to replace a current location andorientation.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A vehicle and trailer maneuvering system or maneuver assist systemand/or driving assist system operates to capture images exterior of thevehicle and trailer being towed by the vehicle and may process thecaptured image data to determine a path of travel for the vehicle andtrailer and to detect objects at or near the vehicle and in thepredicted path of the vehicle, such as to assist a driver of the vehiclein maneuvering the vehicle and trailer in a rearward direction. Thevision system includes an image processor or image processing systemthat is operable to receive image data from one or more cameras and mayprovide an output to a display device for displaying imagesrepresentative of the captured image data. Optionally, the vision systemmay provide display, such as a rearview display or a top down or bird'seye or surround view display or the like.

Referring now to the drawings and the illustrative embodiments depictedtherein, a vehicle 10 includes a trailer maneuver assist system 12 thatis operable to maneuver the vehicle 10 and trailer 14 toward a desiredor selected location. The trailer maneuver assist system 12 includes atleast one exterior viewing vehicle-based imaging sensor or camera, suchas a rearward viewing imaging sensor or camera 16 (and the system mayoptionally include multiple exterior viewing imaging sensors or cameras,such as a sideward/rearward viewing camera 18, 20 at respective sides ofthe vehicle), and a rearward viewing trailer-based camera 22, whichcapture image data representative of the scene exterior of the vehicleand trailer, with the cameras each having a lens for focusing images ator onto an imaging array or imaging plane or imager of the cameras (FIG.1). Optionally, a forward viewing camera may be disposed at thewindshield of the vehicle and view through the windshield and forward ofthe vehicle, such as for a machine vision system (such as for trafficsign recognition, headlamp control, pedestrian detection, collisionavoidance, lane marker detection and/or the like). The trailer maneuverassist system 12 includes a control or electronic control unit (ECU) orprocessor that is operable to process image data captured by the cameraor cameras and may detect objects or the like and/or provide displayedimages at a display device for viewing by the driver of the vehicle (thecontrol and/or display device may be part of or incorporated in or at aninterior rearview mirror assembly of the vehicle, or the control and/orthe display device may be disposed elsewhere at or in the vehicle). Thedata transfer or signal communication from the camera to the ECU maycomprise any suitable data or communication link, such as a vehiclenetwork bus or the like of the equipped vehicle.

The system of the present invention is operable to automaticallymaneuver a trailer which is connected to a vehicle to a user's desiredor selected location. The system receives an input from a touch screenof the vehicle (such as at a center stack display) for the destinationlocation and converts the pixel (X, Y) coordinates from the touch inputto real world coordinates (XYZ). The system plans the path of thevehicle and trailer towards the desired location. The system may supportautonomous driving of the vehicle, and may automatically control thevehicle steering angle to direct the trailer toward the desired orselected location.

After initial selection of the target area displayed on the screen viathe user input, reversing of the vehicle and the trailer to the targetlocation (represented by the selected target area) is at leastsemi-autonomous in that at least steering, and preferably at leaststeering and the reversing speed of the vehicle, and more preferably atleast steering, reversing speed and braking of the vehicle, is underautomatic control by the control system. Further, image and/or sensordata captured by the vehicle and trailer sensors, during suchsemi-autonomous reversing maneuvering of the vehicle and trailer, isprocessed at a data processor using machine vision techniques, such asknown machine vision techniques. And should such data processing detectan object either in the determined or projected path of travel or likelyto enter the determined or projected path of travel (such as anothervehicle or pedestrian or bicyclist or the like present in or approachingthe determined or projected path of travel of the vehicle and trailer),the system reacts to mitigate any potential collision or hazard in suchsituations. For example, based on such a detected potential collision orhazard, an alert may be generated to a driver or occupant of the vehiclealerting that discontinuing the current reversing maneuver would berecommended. Optionally, for example, based on such a detected potentialcollision or hazard, the system may stop the maneuver itself and awaitfurther instructions/input from user. Optionally, for example, based onsuch a detected potential collision or hazard, the system may adjust therearward path of travel to mitigate or avoid impact with or hazard bythe detected object.

The system does not require user measurements and/or a target stickerattached to a trailer. The system may work with any type of conventionaltrailer such as vehicle haulers, box trailers, utility trailers, loadedor unloaded boat trailers, snow mobile trailers, and any other customtrailers. The system functions independent of lighting conditions (suchas, for example, day/night/sun/cloud/rain/snow) and road surfaces (suchas, for example, concrete/asphalt/gravel/grass/dirt). The system mayrequire an initial calibration where the vehicle is driving forward forless than five meters. However, no vehicle motion is preferred (nocalibration).

As shown in FIG. 1, the trailer maneuver assist (TMA) system includes acamera 22 at the rear end of a trailer (trailer camera), and amulti-camera system on the towing vehicle. The multi-camera systemincludes a rearward viewing camera 16 (such as a rear backup camera), aleft camera 18 (such as a side camera disposed at a left side rearviewmirror assembly of the vehicle), a right camera 20 (such as a sidecamera disposed at a left side rearview mirror assembly of the vehicle),and a center high mounted stop lamp (CHMSL) camera 24. The TMA system isoperable to perform without degradation, if at least the trailer cameraand rear view camera on the towing vehicle are present.

The algorithm for the TMA system is divided into three sections orblocks, as shown in FIG. 2. The system receives image data captured bythe cameras, and receives a touch input indicative of the targetlocation for the trailer, and performs an input analysis. Responsive tothe input analysis, the system plans the target path toward the targetlocation. The system includes a trailer control loop to control thevehicle to maneuver the vehicle and trailer toward the target location.The algorithm flow is shown in FIG. 3.

When the trailer maneuver assist (TMA) system is activated at a towingvehicle, the system checks if a trailer is connected to the towingvehicle, and the system checks if the connected trailer is already beencalibrated or prompts a list of previously calibrated trailer to theuser, such as at either an instrument cluster or center stack display ofthe towing vehicle, which allows the user to select a trailer from thelist. If the trailer is new (not previously calibrated), the system mayperform a one-time calibration for the trailer necessary toalign/calibrate the system for that trailer.

If the trailer is not calibrated, the system prompts a message, eitherat the vehicle instrument cluster or center stack display, to the userto calibrate the trailer with instructions displayed at either theinstrument cluster or center stack display to calibrate the trailer. Thecalibration instructions include “drive forward with/without turn”.During the calibration phase, the TMA system finds the relation betweenthe camera installed on the trailer and the towing vehicle, and measurespixel motion vectors (PMVs) of the trailer camera (T_PMV), and measuresthe PMVs for each camera in the multi-camera system of the towingvehicle (VChmsl_PMV, VLeft_PMV, VRight_PMV, VRear_PMV), such as shown inFIG. 5. The TMA system derives correlation matrices (CM[ ]) of thetrailer camera with each and every camera of the towing vehicle'smulti-camera system. Only a one time calibration sequence is requiredfor each new trailer. Once the calibration sequence is completed, theTMA system provides confirmation to the user on either the instrumentcluster or center stack display of the towing vehicle. The TMA systemstores the calibration values of the trailer into non-volatile memory ofthe hardware on which the TMA algorithm and software is running. ThePMVs include information about pixel speed (m/sec) in an orientation,and the maximum correlation matrix for the PMVs (MAX_CM_PMV) includesinformation pertaining to the maximum allowed pixel speed at the maximumallowed relative orientation of two cameras.

If the connected trailer has already been calibrated, or after thetrailer has been calibrated, the system will display the bird's eye viewof “towing vehicle+trailer” in a scene map on the display screen (suchas a display screen at a center stack) of the vehicle. The bird's eyeview or surround view is a top view projected image (comprising imagesderived from image data captured by the cameras of the vehicle andtrailer) that shows 360 degree surroundings of towing vehicle and thetrailer attached to it. The scene map (FIG. 4) is the region behind thetowing vehicle which covers ‘x’ meters along a longitudinal axis fromthe center of the rear view camera and ‘x/2’ meters at each side oflateral axis from the center of the rear view camera. The dimension ‘x’is preferably greater than 2× the length of the trailer, and thedimension ‘x’ depends on image resolution, field of view and height ofthe of the trailer camera. The scene map is generated using the trailercamera and the multi camera system on the towing vehicle. The TMA systemprompts a message to the user (such as at the center stack display) toselect a location in the scene map to which the trailer has to maneuver.The user may select a target location or point (X, Y) on the centerstack display either by touching the screen (if the screen is touchenabled) or by using cursor buttons to move a cursor to the targetlocation or point on the scene map.

When the user selects a location on the scene map to which the towingvehicle is to be maneuvered, the selected location in the screencoordinate system is then transformed into real world location (XYZ)using calibration parameters of the camera(s). The TMA system detectsavailable free space available in the scene map in which the towingvehicle and trailer can be maneuvered, where the free space is free ofobstacles. The TMA system identifies the path to maneuver towards thedesired location. As shown in FIG. 4, the vehicle and trailer in thescene map is shown in the displayed bird's eye view to the user. The TMAsystem initiates the maneuver towards the desired location once the useragrees with the path planned by TMA system. The TMA system stopsmaneuvering the vehicle and provides a message to the user whenever anobstacle enters into the planned path.

When the user selects a location on the scene map, the TMA systemautomatically calculates the distance between the rear end of thetrailer and the user's desired location (D_TrP), the path towards theuser's desired location, and the desired correlation matrix (DCM[ ])between the trailer camera and the vehicle multi-camera system for every1/500^(th) distance of D_TrP using pixel motion vectors (PMVs) of thetrailer camera and the vehicle multi cameras. When the maneuver isinitiated, the TMA system continuously calculates the currentcorrelation matrix (CCM) at a frequency of, for example, 60 samples persecond. If the CCM at a given point of time x is not matching withDCM[x] during the maneuver, then the trailer position will be adjustedby controlling the steering of the towing vehicle until the CCM matcheswith DCM values and the DCM will be updated as per the new position ofthe trailer. This step continues to happen until the vehicle reaches thedesired location. The TMA system considers the direction of maneuver toalign the pixel motion vector in the trailer camera with that of themulti-camera system of the towing vehicle. The system shall not crossthe maximum allowed pixel speed at the maximum allowed relativeorientation of two cameras (MAX_CM_PMV) at any given point of time.

Optionally, an advanced trailer assist (ATA) system may be similar tothe TMA system described above, but may use various sensors at thevehicle and trailer. For example, and such as shown in FIGS. 6 and 6A,the advanced trailer assist (ATA) system includes a plurality of sensors122 around or at the rear end of a trailer (trailer sensors), and amulti-sensor sensing system on the towing vehicle. The multi-sensorsensing system may include a plurality of rearward sensing sensors 116,a plurality of forward sensing sensors 117, a left sensor 118, a rightsensor 120, and a center high mounted stop lamp (CHMSL) sensor 124 (theside sensors and CHMSL sensor may comprise cameras and the forward andrearward sensors and the trailer sensors may comprise radar sensors,lidar sensors and/or ultrasonic sensors and/or the like). The multiplesensors at the trailer (and similarly at the rear of the tow vehicle andat the front/sides of the tow vehicle) may comprise a camera orimage-based sensor and a non-imaging sensor, such as a radar sensor,lidar sensor, an ultrasonic sensor and/or the like, such as shown inFIG. 6.

The algorithm for the ATA system is divided into three sections orblocks, as shown in FIG. 7. The system receives data sensed by thesensors, and receives a user input indicative of the target location forthe trailer, and performs an input analysis. Responsive to the inputanalysis, the system plans the target path toward the target location.The system includes a trailer control loop to control the vehicle tomaneuver the vehicle and trailer toward the target location. Thealgorithm flow is shown in FIG. 8.

When the ATA system is activated at a towing vehicle, the system checksif a trailer is connected to the towing vehicle, and the system checksif the connected trailer is already been calibrated or, if it is acalibrated trailer, the system prompts a list of previously calibratedtrailer to the user, such as at either an instrument cluster or centerstack display of the towing vehicle, which allows the user to select atrailer from the list. If the trailer is new (not previouslycalibrated), the system may perform a one-time calibration for thetrailer necessary to align/calibrate the system for that trailer.

If the trailer is not calibrated, the system prompts a message, eitherat the vehicle instrument cluster or center stack display, to the userto calibrate the trailer with instructions displayed at either theinstrument cluster or center stack display to calibrate the trailer. Thecalibration instructions include “drive forward with/without turn”.During the calibration phase, the TMA system finds the relation betweenthe sensors installed on the trailer and the towing vehicle, andmeasures motion vectors (MVs) of the trailer sensors (T_MV), andmeasures the MV for each sensor in the multi-sensor system of the towingvehicle (VChmsl_MV, VLeft_MV, VRight_MV, VRear_MV). The ATA systemderives correlation matrices (CM[ ]) of the trailer sensors with eachand every sensor of the towing vehicle's multi-sensor system. Only a onetime calibration sequence is required for each new trailer. And minimaleffort is needed to calibrate the trailer with the towing vehicle.

Once the calibration sequence is completed, the ATA system providesconfirmation to the user on either the instrument cluster or centerstack display of the towing vehicle. The ATA system stores thecalibration values of the trailer into non-volatile memory of thehardware on which the ATA algorithm and software is running. The MVsinclude information about relative data change information, and themaximum correlation matrix for the MVs (MAX_CM_MV) includes informationpertaining to the maximum allowed relative data change at the maximumallowed relative orientation between two sensors of same type.

If the connected trailer has already been calibrated, or after thetrailer has been calibrated, the system will present one display thebird's eye view of “towing vehicle+trailer” in a relative scene map onthe display screen (such as a display screen at a center stack) of thevehicle. The relative scene map is the region behind the towing vehiclewhich covers ‘x’ meters along a longitudinal axis from the center of therear sensor and ‘x/2’ meters at each side of lateral axis from thecenter of the rear sensor. The dimension ‘x’ is preferably greater than2× the length of the trailer, and the dimension ‘x’ depends on imageresolution, field of view and height of the of the trailer sensors. Therelative scene map is generated using the trailer sensors and the multisensor system on the towing vehicle. The ATA system prompts a message tothe user (such as at the center stack display) to select a location inthe relative scene map to which the trailer has to maneuver. The usermay select a target location or point (X, Y) on the center stack displayeither by touching the screen (if the screen is touch enabled) or byusing cursor buttons to move a cursor to the target location or point onthe scene map or by otherwise providing an input indicative of thetarget location.

When the user selects a location on the scene map to which the towingvehicle is to be maneuvered, the selected location in the screencoordinate system is then transformed into real world location (XYZ)using calibration parameters of the sensors. The ATA system detectsavailable free space available in the scene map in which the towingvehicle and trailer can be maneuvered, where the free space is free ofobstacles. The ATA system identifies the path to maneuver towards thedesired location. In a similar manner as shown in FIG. 4, the vehicleand trailer in the scene map is shown in the displayed bird's eye viewimages for viewing by the user. The ATA system initiates the maneuvertowards the desired location once the user agrees with the path plannedby the ATA system. The ATA system stops maneuvering the vehicle andprovides a message to the user whenever an obstacle enters into theplanned path.

When the user selects a location on the scene map, the ATA systemautomatically calculates the distance between the rear end of thetrailer and the user's desired location (D_TrP), the path towards theuser's desired location, and the desired correlation matrix (DCM[ ])between the trailer sensors and the vehicle multi-sensor system forevery 1/500^(th) distance of D_TrP using motion vectors (MVs) of thetrailer sensors and the vehicle sensors. When the maneuver is initiated,the ATA system continuously calculates the current correlation matrix(CCM) at a frequency of, for example, 60 samples per second. If the CCMat a given point of time x is not matching with DCM[x] during themaneuver, then the trailer position will be adjusted by controlling thesteering of the towing vehicle until the CCM matches with DCM values andthe DCM will be updated as per the new position of the trailer. Thisstep continues to happen until the vehicle reaches the desired location.The ATA system considers the direction of maneuver to align the relativedata change rate of the trailer sensors with that of the multi-sensorsystem of the towing vehicle. The system shall not cross the maximumallowed speed at the maximum allowed relative orientation of two sensors(MAX_CM_MV) at any given point of time.

The system may use (as inputs) an on/off vehicle navigation system,which directs the trailer to the destination (see FIG. 9). The systemmay support both on vehicle navigation systems (vehicle-installedsystems) and off vehicle navigation systems, such as a portable or handheld navigation device.

Optionally, the system may use a drone camera for an input. For example,the system may use camera images from external sensor data including adrone camera to create the relative scene map for the trailer maneuver.The drone may park or dock at the vehicle or trailer and may detach forremote image data capture above the vehicle and/or trailer and/orsurrounding area, whereby the captured image data may be wirelesslytransmitted to the vehicle system to assist the system in maneuveringthe vehicle and trailer. Optionally, the drone may, when undocked,remain tethered to the vehicle or trailer and may communicate capturedimage data via a wired connection. The system may utilize aspects of thesystems described in U.S. Publication No. US-2018-0141658, which ishereby incorporated herein by reference in its entirety.

Optionally, the system may use a target based Input, such as an activetarget, where a remote device or transmitter may function as an activetarget. When the user activates the system, it shall track and followthe location of the remote device (see FIG. 10).

Optionally, the system may use a passive target, where the system maytrack any user defined external target. The system may use the target asa reference for the destination location of the trailer. For example,the system may use a user defined painting on a garage wall as areference passive target and, upon detecting it, the system maneuversthe vehicle and trailer towards the target.

The system may allow for a user path selection, where a user can definea path for the trailer to maneuver it towards its destination. Thesystem may map the user defined path to the real world coordinatesystem, and will identify the way points to complete the maneuver.Optionally, the system may allow for an orientation selection, where theuser can identify the orientation of the trailer along with defining theposition and the path. Optionally, the system may be responsive to avehicle-to-vehicle/device communication, where the system may usevehicle-to-vehicle (V2V) or vehicle or other device communication as oneof the input methods to identify the obstacles and free space.

The system of the present invention is suitable for use on all types oftrailers, including trailers attached with a ball/hitch, fifth wheeltrailers, pintle trailers and/or goose neck trailers. The system is alsosuitable for trailers used in the agricultural Industry, used oncommercial trucks, tugs, off-road vehicles (ORVs), in the constructionindustry and in military applications. The system is also suitable fornon-trailer applications, and manufacturing vehicles and applicationswhere multiple trailers are attached to a single towing vehicle.

The system thus automatically guides the trailer that is connected tothe towing vehicle to a user's selected target location. The systemtakes the input from the user to select the destination location andconvert the input data or coordinates to real world coordinates (XYZ).The system then plans the path of the vehicle (and trailer) towards thedestination and maneuvers the vehicle (and trailer) along thedetermined/planned path.

Thus, the trailer maneuvering assist (TMA) system and/or advancedtrailer assist (ATA) system of the present invention operates responsiveto a user selecting or setting a location on the display that representsthe location to which the trailer is to be maneuvered. The systemautomatically calculates an obstacle free path from the initial positionof the vehicle and trailer to the selected location. The systemestimates control parameters at particular intervals required toregulate the trailer for a safer maneuver. The system continuouslyupdates the control parameters during the course of maneuvering andadjusts the path accordingly. The system does not require any sticker ortarget on the trailer. For a trailer setup process, the user drives thetowing vehicle in a forward direction with/without a turn. Whenever anew trailer is connected to the towing vehicle, the user needs to go fora calibration drive to set up the trailer for the vehicle. The systemcontrols the accelerator and brakes of the vehicle while it steers thevehicle to keep the vehicle and trailer moving backward along thedetermined path and toward the selected location. The user still shouldmonitor the display/mirror and apply the vehicle brakes or control thevehicle steering if required, such as to avoid an obstacle or change thepath of rearward travel of the vehicle and trailer.

Optionally, after the user has selected a target or destination locationfor the vehicle and trailer, the user may select a new destination. TheATA system thus may provide an option to the user to change or adjustthe initial/current destination location and/or the orientation of atrailer that is attached to the host vehicle. The ATA system provides anoption to the user to change or adjust the initial destination locationand/or orientation of the trailer with or without stopping the maneuverof the host vehicle. The ATA system may utilize one or more of userinput types, such as a wide variety of input types, foradjusting/changing the destination location.

For example, and such as shown in FIG. 11, the system may provide a dragor touch at the display screen to provide selection of a new destinationor target location. The driver may select a first target location ordestination, and the system may provide for a dynamic (adjusted)destination location. The user is then allowed to change or adjust thedestination location, including the orientation of the trailer that isattached to the host vehicle, at any point of time after activating theATA feature before the trailer reaches the current destination locationor upon the trailer arriving at the current destination location.

Once the user adjusts the destination location and orientation of thetrailer, the ATA system automatically updates the path planner andcalculates the vehicle control triggers required in the new plannedpath. The system uses the current position of the trailer to update thepath planner. Options for the vehicle control triggers to direct thetrailer to the new destination location include (a) in case of fullautonomous driving of the vehicle and trailer, braking control,acceleration control, gear change and steering wheel control, or (b) incase of semi-autonomous driving, steering wheel control only.

The dynamic destination option provides several advantages to the enduser, such as (a) collision avoidance of the trailer with obstacles inthe current path by changing or adjusting the path and (b) adjustment ofthe path when the current path is difficult to maneuver. For example,the following are some of the HMI or control options that can be used bythe user to select or choose the fixed or dynamic destinations. The usermay drag or move or touch the screen for the new destination locationand orientation of trailer on touch screen to change the destinationlocation (see FIG. 11). Optionally, soft or hard buttons may be used tofix/adjust/change the current destination location and orientation ofthe trailer. Optionally, a knob (FIG. 12) may be used to select thefixed or adjusted destination location and orientation of the trailer.Optionally, the steering wheel may be used to select the fixed oradjusted destination location and orientation of the trailer when theATA feature is active.

Referring now to FIGS. 13 and 14, the TMA/ATA feature may use camerasand/or sensors on the towing vehicle and/or trailer to create a surroundand/or three-dimensional (3D) views of trailer backup maneuvers. Forexample, the user may be able control or select a location/destinationand orientation of a virtual camera to provide a useful view on adisplay for the backup maneuver. The user may control the virtual camerathrough a variety of control inputs (e.g., touch screens, knobs, etc.).Optionally, a rearward directed spot light may be used to enhancesensing of the arear rearward of the vehicle and trailer during trailerback-up maneuvers, particularly in low ambient lighting conditions ornighttime driving conditions. The ATA may also use guidelines and/oroverlays (2D or 3D) to further assist the user before and during themaneuver. The ATA feature may provide the views in such a manner thatthe user perceives no blind spots around the towing vehicle and/or thetrailer. The system uses the sensors on the towing vehicle and/or on thetrailer to create the views (2D and/or 3D) during the trailer back-upmaneuvers.

Thus, the user may, while the vehicle is reversing toward a previouslyselected target location or destination, select a new destination and/orselect a new target orientation of the trailer at the destination(initial or newly selected). The system, responsive to the new targetlocation or destination or orientation, adjusts the path of travel ofthe vehicle and trailer accordingly so that the vehicle is controlled tomaneuver the trailer toward the newly selected destination. The systemmay further adjust the path of travel for further selections of targetsor destinations and/or for further selections of targeted or desiredorientation(s) of the trailer at the target or destination.

The system may utilize aspects of the trailering or trailer angledetection systems or trailer hitch assist systems described in U.S. Pat.Nos. 9,085,261 and/or 6,690,268, and/or U.S. Publication Nos.US-2018-0215382; US-2017-0254873; US-2017-0217372; US-2017-0050672;US-2015-0217693; US-2014-0160276; US-2014-0085472 and/orUS-2015-0002670, and/or U.S. patent applications, Ser. No. 16/033,416,filed Jul. 12, 2018, now U.S. Pat. No. 10,532,698, Ser. No. 15/928,128,filed Mar. 22, 2018, now U.S. Pat. No. 10,552,976, Ser. No. 15/928,126,filed Mar. 22, 2018, now U.S. Pat. No. 10,885,652, Ser. No. 15/910,100,filed Mar. 2, 2018, now U.S. Pat. No. 10,706,291, which are herebyincorporated herein by reference in their entireties.

The system may utilize aspects of the parking assist systems describedin U.S. Pat. No. 8,874,317 and/or U.S. Publication Nos. US-2017-0329346;US-2017-0317748; US-2017-0253237; US-2017-0050672; US-2017-0017847;US-2017-0015312 and/or US-2015-0344028, which are hereby incorporatedherein by reference in their entireties.

The cameras or sensors may comprise any suitable cameras or sensors.Optionally, one or more of the cameras may comprise a “smart camera”that includes the imaging sensor array and associated circuitry andimage processing circuitry and electrical connectors and the like aspart of a camera module, such as by utilizing aspects of the visionsystems described in International Publication Nos. WO 2013/081984and/or WO 2013/081985, which are hereby incorporated herein by referencein their entireties.

The system includes an image processor operable to process image datacaptured by the camera or cameras, such as for detecting objects orother vehicles or pedestrians or the like in the field of view of one ormore of the cameras. For example, the image processor may comprise animage processing chip selected from the EyeQ family of image processingchips available from Mobileye Vision Technologies Ltd. of Jerusalem,Israel, and may include object detection software (such as the typesdescribed in U.S. Pat. Nos. 7,855,755; 7,720,580 and/or 7,038,577, whichare hereby incorporated herein by reference in their entireties), andmay analyze image data to detect vehicles and/or other objects.Responsive to such image processing, and when an object or other vehicleis detected, the system may generate an alert to the driver of thevehicle and/or may generate an overlay at the displayed image tohighlight or enhance display of the detected object or vehicle, in orderto enhance the driver's awareness of the detected object or vehicle orhazardous condition during a driving maneuver of the equipped vehicle.

The vehicle may include any type of sensor or sensors, such as imagingsensors or radar sensors or lidar sensors or ladar sensors or ultrasonicsensors or the like. The imaging sensor or camera may capture image datafor image processing and may comprise any suitable camera or sensingdevice, such as, for example, a two dimensional array of a plurality ofphotosensor elements arranged in at least 640 columns and 480 rows (atleast a 640×480 imaging array, such as a megapixel imaging array or thelike), with a respective lens focusing images onto respective portionsof the array. The photosensor array may comprise a plurality ofphotosensor elements arranged in a photosensor array having rows andcolumns. Preferably, the imaging array has at least 300,000 photosensorelements or pixels, more preferably at least 500,000 photosensorelements or pixels and more preferably at least 1 million photosensorelements or pixels. The imaging array may capture color image data, suchas via spectral filtering at the array, such as via an RGB (red, greenand blue) filter or via a red/red complement filter or such as via anRCC (red, clear, clear) filter or the like. The logic and controlcircuit of the imaging sensor may function in any known manner, and theimage processing and algorithmic processing may comprise any suitablemeans for processing the images and/or image data.

For example, the vision system and/or processing and/or camera and/orcircuitry may utilize aspects described in U.S. Pat. Nos. 9,233,641;9,146,898; 9,174,574; 9,090,234; 9,077,098; 8,818,042; 8,886,401;9,077,962; 9,068,390; 9,140,789; 9,092,986; 9,205,776; 8,917,169;8,694,224; 7,005,974; 5,760,962; 5,877,897; 5,796,094; 5,949,331;6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202;6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452;6,822,563; 6,891,563; 6,946,978; 7,859,565; 5,550,677; 5,670,935;6,636,258; 7,145,519; 7,161,616; 7,230,640; 7,248,283; 7,295,229;7,301,466; 7,592,928; 7,881,496; 7,720,580; 7,038,577; 6,882,287;5,929,786 and/or 5,786,772, and/or U.S. Publication Nos.US-2014-0340510; US-2014-0313339; US-2014-0347486; US-2014-0320658;US-2014-0336876; US-2014-0307095; US-2014-0327774; US-2014-0327772;US-2014-0320636; US-2014-0293057; US-2014-0309884; US-2014-0226012;US-2014-0293042; US-2014-0218535; US-2014-0218535; US-2014-0247354;US-2014-0247355; US-2014-0247352; US-2014-0232869; US-2014-0211009;US-2014-0160276; US-2014-0168437; US-2014-0168415; US-2014-0160291;US-2014-0152825; US-2014-0139676; US-2014-0138140; US-2014-0104426;US-2014-0098229; US-2014-0085472; US-2014-0067206; US-2014-0049646;US-2014-0052340; US-2014-0025240; US-2014-0028852; US-2014-005907;US-2013-0314503; US-2013-0298866; US-2013-0222593; US-2013-0300869;US-2013-0278769; US-2013-0258077; US-2013-0258077; US-2013-0242099;US-2013-0215271; US-2013-0141578 and/or US-2013-0002873, which are allhereby incorporated herein by reference in their entireties. The systemmay communicate with other communication systems via any suitable means,such as by utilizing aspects of the systems described in InternationalPublication Nos. WO/2010/144900; WO 2013/043661 and/or WO 2013/081985,and/or U.S. Pat. No. 9,126,525, which are hereby incorporated herein byreference in their entireties.

The system may utilize sensors, such as radar or lidar sensors or thelike. The sensing system may utilize aspects of the systems described inU.S. Pat. Nos. 9,753,121; 9,689,967; 9,599,702; 9,575,160; 9,146,898;9,036,026; 8,027,029; 8,013,780; 6,825,455; 7,053,357; 7,408,627;7,405,812; 7,379,163; 7,379,100; 7,375,803; 7,352,454; 7,340,077;7,321,111; 7,310,431; 7,283,213; 7,212,663; 7,203,356; 7,176,438;7,157,685; 6,919,549; 6,906,793; 6,876,775; 6,710,770; 6,690,354;6,678,039; 6,674,895 and/or 6,587,186, and/or International PublicationNos. WO 2018/007995 and/or WO 2011/090484, and/or U.S. Publication Nos.US-2018-0045812; US-2018-0015875; US-2017-0356994; US-2017-0315231;US-2017-0276788; US-2017-0254873; US-2017-0222311 and/orUS-2010-0245066, and/or U.S. patent application Ser. No. 15/897,268,filed Feb. 15, 2018, now U.S. Pat. No. 10,782,388, which are herebyincorporated herein by reference in their entireties.

Optionally, the vision system may include a display for displayingimages captured by one or more of the imaging sensors for viewing by thedriver of the vehicle while the driver is normally operating thevehicle. Optionally, for example, the vision system may include a videodisplay device, such as by utilizing aspects of the video displaysystems described in U.S. Pat. Nos. 5,530,240; 6,329,925; 7,855,755;7,626,749; 7,581,859; 7,446,650; 7,338,177; 7,274,501; 7,255,451;7,195,381; 7,184,190; 5,668,663; 5,724,187; 6,690,268; 7,370,983;7,329,013; 7,308,341; 7,289,037; 7,249,860; 7,004,593; 4,546,551;5,699,044; 4,953,305; 5,576,687; 5,632,092; 5,708,410; 5,737,226;5,802,727; 5,878,370; 6,087,953; 6,173,501; 6,222,460; 6,513,252 and/or6,642,851, and/or U.S. Publication Nos. US-2014-0022390;US-2012-0162427; US-2006-0050018 and/or US-2006-0061008, which are allhereby incorporated herein by reference in their entireties. Optionally,the vision system (utilizing a forward viewing camera and a rearwardviewing camera and other cameras disposed at the vehicle with exteriorfields of view) may be part of or may provide a display of a top-downview or bird's-eye view system of the vehicle or a surround view at thevehicle, such as by utilizing aspects of the vision systems described inInternational Publication Nos. WO 2010/099416; WO 2011/028686; WO2012/075250; WO 2013/019795; WO 2012/075250; WO 2012/145822; WO2013/081985; WO 2013/086249 and/or WO 2013/109869, and/or U.S.Publication No. US-2012-0162427, which are hereby incorporated herein byreference in their entireties.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent lawincluding the doctrine of equivalents.

The invention claimed is:
 1. A trailer assist system for a vehicle, saidtrailer assist system comprising: a plurality of vehicle sensorsdisposed at the vehicle, said plurality of vehicle sensors comprising arear vehicle sensor disposed at a rear portion of the vehicle and havinga field of sensing at least rearward of the vehicle, a driver sidevehicle sensor disposed at a driver side of the vehicle and having afield of sensing at least sideward of the vehicle, and a passenger sidevehicle sensor disposed at a passenger side of the vehicle and having afield of sensing at least sideward of the vehicle, wherein at least onesensor of said plurality of vehicle sensors comprises a camera; at leastone trailer sensor disposed at a rear portion of a trailer and having afield of sensing at least rearward of the trailer; a control comprisinga processor operable to process data captured by said plurality ofvehicle sensors; a display screen disposed in the vehicle and viewableby an occupant of the vehicle, wherein said display screen displaysimages derived from data captured by at least some of said plurality ofvehicle sensors, wherein the displayed images comprise images of atleast an area rearward of the trailer; wherein said control receives atarget based input from a transmitter remote from the vehicle when thetransmitter is located at a target location rearward of the trailer;wherein, responsive to receiving the target based input that isrepresentative of the target location rearward of the trailer, saidcontrol, via processing of data captured by said vehicle sensors andsaid at least one trailer sensor, is operable to determine a path oftravel for the vehicle and trailer to follow to maneuver the vehicle andtrailer toward the target location; wherein, responsive to determinationof the path of travel, said control displays the determined path oftravel on the display screen; wherein the transmitter moves relative tothe target location, and wherein the control tracks the location of thetransmitter and updates the target location and the determined path oftravel as the transmitter moves relative to the target location; andwherein said control, responsive to determination of the path of travelfor the vehicle and trailer to follow and responsive to the updateddetermined path of travel and without further user input, controls atleast steering of the vehicle to maneuver the vehicle and trailer alongthe updated determined path of travel toward the target location.
 2. Thetrailer assist system of claim 1, wherein said control is operable, viaprocessing by said processor of data captured by said plurality ofvehicle sensors and said at least one trailer sensor, to detect at leastone object present in the determined path of travel.
 3. The trailerassist system of claim 1, wherein said display screen comprises a touchscreen, and wherein a user input is provided by the occupant of thevehicle touching a target area present in a displayed image at the touchscreen, and wherein said control, responsive to the user input, updatesthe target location and the determined path of travel according to theuser input.
 4. The trailer assist system of claim 1, wherein a userinput is provided by the occupant of the vehicle moving a cursor at thedisplay screen to a target area present in a displayed image at thedisplay screen, and wherein said control, responsive to the user input,updates the target location and the determined path of travel accordingto the user input.
 5. The trailer assist system of claim 1, wherein saidcontrol estimates control parameters at particular intervals along thedetermined path of travel.
 6. The trailer assist system of claim 5,wherein said control updates the estimated control parameters whilemaneuvering the vehicle and trailer along the determined path of traveland adjusts the determined path of travel accordingly.
 7. The trailerassist system of claim 1, wherein each vehicle sensor of said pluralityof vehicle sensors disposed at the vehicle comprise cameras.
 8. Thetrailer assist system of claim 7, wherein said at least one trailersensor comprises a camera.
 9. The trailer assist system of claim 1,wherein said at least one trailer sensor comprises a plurality oftrailer sensors.
 10. The trailer assist system of claim 9, wherein atleast one of said plurality of trailer sensors comprises a camera and atleast one other of said trailer sensors comprises a non-imaging sensor.11. The trailer assist system of claim 9, comprising a plurality ofsensors disposed at a front portion of the vehicle and having a field ofsensing forward of the vehicle.
 12. The trailer assist system of claim1, further comprising a remote sensor in wireless communication withsaid control, wherein said remote sensor is remote from the vehicle andthe trailer, and wherein said control is operable to process datacaptured by said remote sensor while maneuvering the vehicle and traileralong the determined path of travel.
 13. The trailer assist system ofclaim 12, wherein the remote sensor comprises a drone sensor disposed ata drone that is operable to maneuver over the area rearward of thetrailer.
 14. The trailer assist system of claim 1, wherein, responsiveto a user input selecting a second target area present in the displayedimage, said control, via processing of data captured by said vehiclesensors and said trailer sensor, is operable to determine a second pathof travel for the vehicle and trailer to follow to maneuver the vehicleand trailer toward a second physical target location represented by thesecond selected target area, and wherein said control, responsive todetermination of the second path of travel, controls the steering andbraking of the vehicle to maneuver the vehicle and trailer along thedetermined second path of travel toward the second physical targetlocation represented by the displayed second target area.
 15. A trailerassist system for a vehicle, said trailer assist system comprising: aplurality of vehicle cameras disposed at the vehicle, said plurality ofvehicle cameras comprising a rear vehicle camera disposed at a rearportion of the vehicle and having a field of view at least rearward ofthe vehicle, a driver side vehicle camera disposed at a driver side ofthe vehicle and having a field of view at least sideward of the vehicle,and a passenger side vehicle camera disposed at a passenger side of thevehicle and having a field of view at least sideward of the vehicle; atrailer camera disposed at a rear portion of a trailer and having afield of view at least rearward of the trailer; a control comprising aprocessor operable to process image data captured by said plurality ofvehicle cameras; wherein said control receives a target based input froma transmitter remote from the vehicle when the transmitter is located ata target location rearward of the trailer; wherein, responsive toreceiving the target based input that is representative of the targetlocation rearward of the trailer, said control, via processing of imagedata captured by said plurality of vehicle cameras and said trailercamera, is operable to determine a path of travel for the vehicle andtrailer to follow to maneuver the vehicle and trailer toward the targetlocation; wherein the transmitter moves relative to the target location,and wherein the control tracks the location of the transmitter andupdates the target location and the determined path of travel as thetransmitter moves relative to the target location; and wherein saidcontrol, responsive to determination of the path of travel for thevehicle and trailer to follow and responsive to the updated determinedpath of travel, and without further user input, controls at leaststeering of the vehicle to maneuver the vehicle and trailer along thedetermined path of travel toward the target location.
 16. The trailerassist system of claim 15, wherein the transmitter is operable totransmit the target based input responsive to actuation of a user inputof the transmitter.
 17. A trailer assist system for a vehicle, saidtrailer assist system comprising: a plurality of vehicle camerasdisposed at the vehicle, said plurality of vehicle cameras comprising arear vehicle camera disposed at a rear portion of the vehicle and havinga field of view at least rearward of the vehicle, a driver side vehiclecamera disposed at a driver side of the vehicle and having a field ofview at least sideward of the vehicle, and a passenger side vehiclecamera disposed at a passenger side of the vehicle and having a field ofview at least sideward of the vehicle; a trailer camera disposed at arear portion of a trailer hitched to the vehicle and having a field ofview at least rearward of the trailer; a control comprising a processoroperable to process image data captured by said plurality of vehiclecameras and said trailer camera; a display screen disposed in thevehicle and viewable by an occupant of the vehicle, wherein said displayscreen displays images derived from image data captured by at least someof said plurality of vehicle cameras and said trailer camera, whereinthe displayed images comprise images of at least an area rearward of thetrailer; wherein said control receives a target based input from atransmitter remote from the vehicle when the transmitter is located at atarget location rearward of the trailer; wherein, responsive toreceiving the target based input that is representative of the targetlocation rearward of the trailer, said control is operable to determineif the system is calibrated for the trailer hitched to the vehicle;wherein, responsive to determining that the system is not calibrated forthe trailer hitched to the vehicle, said control calibrates the systemfor the trailer; wherein, after the system is calibrated for thetrailer, said control, via processing of image data captured by saidplurality of vehicle cameras and said trailer camera, and responsive toreceiving the target based input that is representative of the targetlocation rearward of the trailer, determines a path of travel for thevehicle and trailer to follow to maneuver the vehicle and trailer towardthe target location; wherein the transmitter moves relative to thetarget location, and wherein the control tracks the location of thetransmitter and updates the target location and the determined path oftravel as the transmitter moves relative to the target location; andwherein, responsive to determination of the path of travel for thevehicle and trailer to follow and responsive to the updated determinedpath of travel, said control displays the determined path of travel onthe display screen; and wherein said control, responsive to confirmationof the determined path of travel from a user, and without further userinput, controls at least steering of the vehicle to maneuver the vehicleand trailer along the determined path of travel toward the targetlocation.
 18. The trailer assist system of claim 17, wherein saidcontrol is operable to generate driving instructions for a driver of thevehicle to follow to calibrate the system for the trailer hitched to thevehicle.
 19. The trailer assist system of claim 17, wherein said controlis operable to measure motion vectors of at least one of the pluralityof vehicle cameras and the trailer camera to calibrate the system forthe trailer hitched to the vehicle.